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Abstract 



Multi-agent planning in stochastic environments can be framed formally as a decen- 
tralized Markov decision problem. Many rcal-lifc distributed problems that arise in manu- 
facturing, multi-robot coordination and information gathering scenarios can be formalized 
using this framework. However, finding the optimal solution in the general case is hard, 
limiting the applicability of recently developed algorithms. This paper provides a practi- 
cal approach for solving decentralized control problems when communication among the 
decision makers is possible, but costly. We develop the notion of communication-based 
mechanism that allows us to decompose a decentralized MDP into multiple single-agent 
problems. In this framework, referred to as decentralized semi-Markov decision process 
with direct communication (Dec-SMDP-Com), agents operate separately between commu- 
nications. We show that finding an optimal mechanism is equivalent to solving optimally a 
Dec-SMDP-Com. We also provide a heuristic search algorithm that converges on the opti- 
mal decomposition. Restricting the decomposition to some specific types of local behaviors 
reduces significantly the complexity of planning. In particular, we present a polynomial- 
time algorithm for the case in which individual agents perform goal-oriented behaviors 
between communications. The paper concludes with an additional tractable algorithm 
that enables the introduction of human knowledge, thereby reducing the overall problem 
to finding the best time to communicate. Empirical results show that these approaches 
provide good approximate solutions. 

1. Introduction 

The decentralized Markov decision process has become a common formal tool to study 
multi-agent planning and control from a decision-theoretic perspective (Bernstein, Givan, 
Immerman, & Zilberstein, 2002; Becker, Zilberstein, Lesser, &; Goldman, 2004; Guestrin &: 
Gordon, 2002; Guestrin, Roller, & Parr, 2001; Nair, Tambe, Yokoo, Pynadath, & Marsella, 
2003; Petrik & Zilberstein, 2007; Peshkin, Kim, Meuleau, & Kaelbling, 2000). Seuken 
and Zilberstein (2008) provide a comprehensive comparison of the existing formal models 
and algorithms. Decentralized MDPs complement existing approaches to coordination of 
multiple agents based on on-line learning and heuristic approaches (Wolpert, Wheeler, & 
Tumer, 1999; Schneider, Wong, Moore, k. Riedmiller, 1999; Xuan, Lesser, k. Zilberstein, 
2001; Ghavamzadeh & Mahadevan, 2004; Nair, Tambe, Roth, k Yokoo, 2004). 

Many challenging real-world problems can be formalized as instances of decentralized 
MDPs. In these problems, exchanging information constantly between the decision makers 
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is either undesirable or impossible. Furthermore, these processes are controlled by a group 
of decision makers that must act based on different partial views of the global state. Thus, a 
centralized approach to action selection is infeasible. For example, exchanging information 
with a single central controller can lead to saturation of the communication network. Even 
when the transitions and observations of the agents are independent, the global problem 
may not decompose into separate, individual problems, thus a simple parallel algorithm 
may not be sufficient. Choosing different local behaviors could lead to different global 
rewards. Therefore, agents may need to exchange information periodically and revise their 
local behaviors. One important point to understand the model we propose is that although 
eventually each agent will behave following some local behavior, choosing among possible 
behaviors requires information from other agents. We focus on situations in which this 
information is not freely available, but it can be obtained via communication. 

Solving optimally a general decentralized control problem has been shown to be com- 
putationally hard (Bernstein ct al., 2002; Pynadath &: Tambe, 2002). In the worst case, 
the general problem requires a double-exponential algorithm^. This difficulty is due to two 
main reasons: 1) none of the decision-makers has full-observability of the global system and 
2) the global performance of the system depends on a global reward, which is affected by 
the agents' behaviors. In our previous work (Goldman & Zilberstein, 2004a), we have stud- 
ied the complexity of solving optimalty certain classes of Dcc-MDPs and Dcc-POMDPs^. 
For example, we have shown that decentralized problems with independent transitions and 
observations are considerably easier to solve, namely, they are NP-complete. Even in these 
cases, agents' behaviors can be dependent through the global reward function, which may 
not decompose into separate local reward functions. The latter case has been studied within 
the context of auction mechanisms for weakly coupled MDPs by Bererton et al. (2003). 
In this paper, the solution to this type of more complex decentralized problems includes 
temporally abstracted actions combined with communication actions. Petrik and Zilber- 
stein (2007) have recently presented an improved solution to our previous Coverage Set 
algorithm (Becker et al., 2004), which can solve decentralized problems optimally. How- 
ever, the technique is only suitable when no communication between the agents is possible. 
Another recent study by Seukcn and Zilberstein (2007a, 2007b) produced a more general 
approximation technique based on dynamic programming and heuristic search. While the 
approach shows better scalability, it remains limited to relatively small problems compared 
to the decomposition method presented here. 

We propose an approach to approximate the optimal solutions of decentralized problems 
off-line. The main idea is to compute multiagent macro actions that necessarily end with 
communication. Assuming that communication incurs some cost, the communication policy 
is computed optimally, that is the algorithms proposed in this paper will compute the 
best time for the agents to exchange information. At these time points, agents attain full 
knowledge of the current global state. These algorithms also compute for each agent what 
domain actions to perform between communication, these are temporally abstracted actions 
that can be interrupted at any time. Since these behaviors are computed for each agent 

1. Unless NEXP is different from EXP, we cannot prove the super-exponential complexity. But, it is 
generally believed that NEXP-complete problems require double-exponential time to solve optimally. 

2. In Dcc-MDPs, the observations of all the agents are sufficient to determine the global state, while in 
Dec-POMDPs the global state cannot be fully determined by the observations. 
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separately and independently from each other, the final complete solution of communication 
and action policies is not guaranteed to be globally optimal. We refer to this approach 
as a communication-based decomposition mechanism: the algorithms proposed compute 
mechanisms to decompose the global behavior of the agents into local behaviors that are 
coordinated by communication. Throughout the paper, these algorithms differ in the space 
of behaviors in which they search: our solutions range from the most general search space 
available (leading to the optimal mechanism) to more restricted sets of behaviors. 

The contribution of this paper is to provide a tractable method, namely communication- 
based decomposition mechanisms, to solve decentralized problems, for which no efficient 
algorithms currently exist. For general decentralized problems, our approach serves as a 
practical way to approximate the solution in a systematic way. We also provide an analysis 
about the bounds of these approximations when the local transitions are not independent. 
For specific cases, like those with independent transitions and observations, we show how 
to compute the optimal decompositions into local behaviors and optimal policies of com- 
munication to coordinate the agents' behaviors at the global level. 

Section 3 introduces the notion of communication-based mechanisms. We formally frame 
this approach as a decentralized semi-Markov decision process with direct communication 
(Dcc-SMDP-Com) in Section 4. Section 5 presents the decentralized multi-step backup 
policy-iteration algorithm that returns the optimal decomposition mechanism when no re- 
strictions are imposed on the individual behaviors of the agents. Due to this generality, 
the algorithm is applicable in some limited domains. Section 6 presents a more practical 
solution, considering that each agent can be assigned local goal states. Assuming local 
goal-oriented behavior reduces the complexity of the problem to polynomial in the num- 
ber of states. Empirical results (Section 6.2) support these claims. Our approximation 
mechanism can also be applied when the range of possible local behaviors are provided 
at design time. Since these predetermined local behaviors alone may not be sufficient to 
achieve coordination, agents still need to decide when to communicate. Section 7 presents 
a polynomial-time algorithm that computes the policy of communication, given local poli- 
cies of domain actions. The closer the human-designed local plans are to local optimal 
behaviors, the closer our solution will be to the optimal joint solution. Empirical results for 
the Meeting under Uncertainty scenario (also known as the Gathering Problem in robotics, 
Suzuki and Yamashita, 1999) are presented in Section 7.1. We conclude with a discussion 
of the contributions of this work in Section 8. 

2. The Dec-MDP model 

Previous studies have shown that decentralized MDPs in general are very hard to solve 
optimally and off-line even when direct communication is allowed (Bernstein et al., 2002; 
Pynadath & Tambe, 2002; Goldman & Zilberstein, 2004a). A comprehensive complexity 
analysis of solving optimally decentralized control problems revealed the sources of diffi- 
culty in solving these problems (Goldman k, Zilberstein, 2004a). Very few algorithms were 
proposed that can actually solve some classes of problems optimally and efficiently. 

Wc define a general underlying process which allows agents to exchange messages directly 
with each other as a decentralized POMDP with direct communication: 
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Definition 1 (Dec-POMDP-Com) A decentralized partially-observable Markov decision 
process with direct communication, Dec-POMDP-Com is given by the following tuple: 
M =< S, Ai,A2,T,, Cj:,P, R, Oi, ^2, 0, T >, where 

• S is a finite set of world states, that are factored and include a distinguished initial 

state s^. 

• A\ and A2 are finite sets of actions, ai denotes the action performed by agent i. 

• S denotes the alphabet of messages and fij G S represents an atomic message sent by 
agent i (i.e., ai is a letter in the language). 

• Cs is the cost of transmitting an atomic message: Cs : S — >■ 3f?. The cost of transmit- 
ting a null message is zero. 

• P is the transition probability function. P{s'\s, 01,02) is the probability of moving 
from state s £ S to state s' £ S when agents 1 and 2 perform actions ai and 02 
respectively. This transition model is stationary, i.e., it is independent of time. 

• R is the global reward function. R(s,ai,a2, s') represents the reward obtained by the 
system as a whole, when agent 1 executes action ai and agent 2 executes action 02 in 
state s resulting in a transition to state s' . 

• J7i and J72 are finite sets of observations. 

• O is the observation function. 0{oi,02\s,ai,a2,s') is the probability of observing oi 
and 02 (respectively by the two agents) when in state s agent 1 takes action ai and 
agent 2 takes action a2, resulting is state s' . 

• If the Dec-POMDP has a finite horizon, it is represented by a positive integer T. The 
notation r represents the set of discrete time points of the process. 

The optimal solution of such a decentralized problem is a joint policy that maximizes 
some criteria-in our case, the expected accumulated reward of the system. A joint policy 
is a tuple composed of local policies for each agent, each composed of a policy of action 
and a policy of communication: i.e., a joint policy 5 = {81,82), where (5^ : x S* — t- A^ 
and (Jj? : il| X S* — )• S. That is, a local policy of action assigns an action to any possible 
sequence of local observations and messages received. A local policy of communication 
assigns a message to any possible sequence of observations and messages received. In each 
cycle, agents can perform a domain action, then perceive an observation and then can send 
a message. 

We assume that the system has independent observations and transitions (see Section 6.3 
for a discussion on the general case). Given factored system states s = (si,S2) € S, 
the domain actions a,i and the observations Oj for each agent, the formal definitions^ for 
decentralized processes with independent transitions, and observations follow. We note 
that this class of problems is not trivial since the reward of the system is not necessarily 
independent. For simplicity, we present our definitions for the case of two agents. However, 
the approach presented in the paper is applicable to systems with n agents. 



3. These definitions are based on Goldman and Zilberstein (2004a). We include them here to make the 
paper self-contained. 
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Definition 2 (A Dec-POMDP with Independent Transitions) A Dec-POMDP has 
independent transitions if the set S of states can be factored into two components S = 81x82 
such that: 

Vsi , s'l G 81 , Vs2 , s'2 G 52 , Vai G Ai , Va2 G ^2, 
Pr(s'il(si,S2),ai,a2,S2) = Pr{s'i\si,ai) A 
Pr(4|(si,S2),ai,a2,4) = -Pr(4|s2, 02)- 

In other words, the transition probability P of the Dec-POMDP can be represented as 
P = Pi ■ P2, where Pi = Pr{s'i\si,ai) and P2 = Pr(s2|s2, 02). 

Definition 3 (A Dec-POMDP with Independent Observations) A Dec-POMDP has 
independent observations if the set 8 of states can be factored into two components 8 = 
S'l X S'2 such that: 

VoiGr2i,Vo2Gr22,Vs = (si,S2),s' = (si,S2) ^ S',VaieAi,Va2G^2, 
Pr(oi|(si,S2), 01,02, (4,S2)>02) = Pr{ol\sl,al,s'l)^ 

Pr{o2\{si,S2),ai,a2, (s'l, S2)> oi) = Pr{o2\s2, 02, 4) 

0(oi, 02|(si, S2), ai, 02, (s'l, s'2)) = Pr{oi\{si, S2), ai, 02, {s\, 4), 02)--P?'(o2|(si, ^2), ci, 02, {s\, s'^ 

In other words, the observation probability O of the Dec-POMDP can be decomposed into 
two observation probabilities Oi and O2, such that Oi = Pr(oi|(si, S2), ai, 02, (s'l, ■S2)> ^2) 
and O2 = Pr{o2\isi,S2),ai,a2, {s[,s'2),oi). 

Definition 4 (Dec-MDP) A decentralized Markov decision process (Dec-MDP) is a Dec- 
POMDP, which is jointly fully observable, i.e., the combination of both agents' observations 
determine the global state of the system. 

In previous work (Goldman &; Zilberstein, 2004a), we proved that Dec-MDPs with inde- 
pendent transitions and observations are locally fully-observable. In particular, we showed 
that exchanging the last observation is sufficient to obtain complete information about the 
current global state and it guarantees optimality of the solution. 

We focus on the computation of the individual behaviors of the agents taking into 
account that they can exchange information from time to time. The following sections 
present the communication-based decomposition approximation method to solve Dec-MDPs 
with direct communication and independent transitions and observations. 

3. Communication-based Decomposition Mechanism 

We are interested in creating a mechanism that will tell us what individual behaviors are 
the most beneficial in the sense that these behaviors taken jointly will result in a good 
approximation of the optimal decentralized solution of the global system. Notice that 
even when the system has a global objective, it is not straightforward to compute the 
individual behaviors. The decision problem that requires the achievement of some global 
objective does not tell us which local goals each decision maker needs to reach in order to 
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maximize the value of a joint policy that reaches the global objective. Therefore, we propose 
communication-based decomposition mechanisms as a practical approach for approximating 
the optimal joint policy of decentralized control problems. Our approach will produce two 
results: 1) a set of temporarily abstracted actions for each global state and for each agent 
and 2) a policy of communication, aimed at synchronizing the agents' partial information 
at the time that is most beneficial to the system. 

Formally, a communication-based decomposition mechanism CDM is a function from 
any global state of the decentralized problem to two single agent behaviors or policies: 
CDM : S — )• {Opti,Opt2)- In general, a mechanism can be applied to systems with n 
agents, in which case the decomposition of the decentralized process will be into n individ- 
ual behaviors. In order to study communication-based mechanisms, we draw an analogy 
between temporary and local policies of actions and options. Options were defined by 
Sutton et al. (1999) as temporally abstracted actions, formalized as triplets including a 
stochastic single-agent policy, a termination condition, and a set of states in which they can 
be initiated: opt =<7r : S x A ^ [0, 1], /3 : 5+ ^ [0, 1], / C S'>. An option is available in a 
state s if s G /. 

Our approach considers options with terminal actions (instead of terminal states) . Ter- 
minal actions were also considered by Hansen and Zhou (2003) in the framework of indefinite 
POMDPs. We denote the domain actions of agent i as Ai. The set of terminal actions only 
includes the messages in S. For one agent, an option is given by the following tuple: 
opti =< TT : Si X T — > Ai[jH,I C Si >, i.e., an option is a non-stochastic policy from 
the agent's partial view (local states) and time to the set of its primitive domain actions 
and terminal actions. The local states Si are given by the factored representation of the 
Dec-MDP with independent transitions and observations. Similarly, the transitions between 
local states are known since P(s'|s, ai, 02) = Pi{s'i\si,ai) ■ -P2(s2l*2, 02)- 

In this paper, we concentrate on terminal actions that are necessarily communication 
actions. We assume that all options are terminated whenever at least one of the agents 
initiates communication (i.e., the option of the message sender terminates when it com- 
municates and the hearer's option terminates due to this external event). We also assume 
that there is joint exchange of messages, i.e., whenever one agent initiates communication, 
the global state of the system is revealed to all the agents receiving those messages: when 
agent 1 sends its observation oi to agent 2, it will also receive agent 2's observation 02. 
This exchange of messages will cost the system only once. Since we focus on finite-horizon 
processes, the options may also be artificially terminated if the time limit of the problem is 
reached. The cost of communication Cy, may include, in addition to the actual transmission 
cost, the cost resulting from the time it takes to compute the agents' local policies. 

Communication-based decomposition mechanisms enable the agents to operate sepa- 
rately for certain periods of time. The question, then, is how to design mechanisms that 
will approximate best the optimal joint policy of the decentralized problem. We distinguish 
between three cases: general options, restricted options, and predefined options. 

General options are built from any primitive domain action and communication action 
given by the model of the problem. Searching over all possible pairs of local single-agent 
policies and communication policies built from these general options will lead to the best 
approximation. It is obtained when we compute the optimal mechanism among all possible 
mechanisms. Restricted options limit the space of feasible options to a much smaller set de- 
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fined using certain behavior characteristics. Consequently, we can obtain mechanisms with 
lower complexity. Such tractable mechanisms provide approximation solutions to decen- 
tralized problems for which no efficient algorithms currently exist. Obtaining the optimal 
mechanism for a certain set of restricted options (e.g., goal-oriented options) becomes feasi- 
ble, as we show in Sections 4-6. Furthermore, sometimes, we may consider options that are 
pre-defined. For example, knowledge about effective individual procedures may already ex- 
ist. The mechanism approach allows us to combine such domain knowledge into the solution 
of a decentralized problem. In such situations, where a mapping between global states and 
single-agent behaviors already exists, the computation of a mechanism returns the policy of 
communication at the meta-level of control that synchronizes the agents' partial informa- 
tion. In Section 7, we study a greedy approach for computing a policy of communication 
when knowledge about local behaviors is given. 

Practical concerns lead us to the study of communication-based decomposition mech- 
anisms. In order to design applicable mechanisms, two desirable properties need to be 
considered: 

• Computational complexity — The whole motivation behind the mechanism ap- 
proach is based on the idea that the mechanism itself has low computational com- 
plexity. Therefore, the computation of the CDM mapping should be practical in the 
sense that individual behaviors of each agent will have complexity that is lower than 
the complexity of the decentralized problem with free communication. There is a 
trade-off between the complexity of computing a mechanism and the global reward 
of the system. There may not be a simple way to split the decentralized process into 
separate local behaviors. The complexity characteristic should be taken into account 
when designing a mechanism; different mechanisms can be computed at different levels 
of difficulty. 

• Dominance — A mechanism CDM\ dominates another mechanism CDM2 if the 
global reward attained by CD Mi with some policy of communication is larger than 
the global reward attained by CDM2 with any communication policy. A mechanism 
is optimal for a certain problem if there is no mechanism that dominates it. 

4. Decentralized Semi-Markov Decision Problems 

Solving decentralized MDP problems with a communication-based decomposition mecha- 
nism translates into computing the set of individual and temporally abstracted actions that 
each agent will perform together with a policy of communication that stipulates when to 
exchange information. Hereafter, we show how the problem of computing a mechanism can 
be formalized as a semi-Markov decision problem. In particular, the set of basic actions 
of this process is composed of the temporally abstracted actions together with the com- 
munication actions. The rest of the paper presents three algorithms aimed at solving this 
semi-Markov problem optimally. The algorithms differ in the sets of actions available to the 
decision-makers, affecting significantly the complexity of finding the decentralized solution. 
It should be noted that the optimality of the mechanism computed is conditioned on the 
assumptions of each algorithm (i.e., the first algorithm provides the optimal mechanism over 
all possible options, the second algorithm provides the optimal mechanism when local goals 
are assumed, and the last algorithm computes the optimal policy of communication assum- 
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ing that the local behaviors are given). Formally, a decentralized semi-Markov decision 
problem with direct communication (Dec-SMDP-Com) is given as follows: 

Definition 5 (Dec-SMDP-Com) A factored, finite-horizon Dec-SMDP-Com over an un- 
derlying Dec-MDP-Com M is a tuple 
< M, Opti,Opt2, P'^,R^ > where: 

• S, T,, Cy,, ^i, ^2, P, O and T are components of the underlying process M defined 
in definitions 4 o,nd 1. 

• Opti is the set of actions available to agent i. It comprises the possible options that 
agent i can choose to perform, which terminate necessarily with a communication act: 
opti =< TT : Si X T AiijY^, I (1 Si > . 

• P^ {s', t-\-N\s, t, opti,opt2) is the probability of the system reaching state s' after exactly 
N time units, when at least one option terminates (necessarily with a communication 
act). This probability function is given as part of the model for every value of N, such 
that t-\- N < T . In this framework, after N time steps at least one agent initiates 
communication (for the first time since time t ) and this interrupts the option of the 
hearer agent. Then, both agents get full observability of the synchronized state. Since 
the decentralized process has independent transitions and observations, P^ is the prob- 
ability that either agent has communicated or both of them have. The probability that 
agent i terminated its option exactly at time t -\- N, Pf^ , is given as follows: 

1 if (7ropt.(si,t) e S) A (A^ = l) A {s'^ = s,)) 

if{noptM,t) e E)A(iV = l)A(s^^Si)) 

if (7rojrt,(si,i) e A)A(iV = l)) 

if (7rop(,(si,i) e S) A(7V>1)) 

if (7ropt,(si,i) e A)A(iV>l)) 
Eg,eS,Pi{Qi\si,TroptM,t))Pl'{s'i,{t+l) + {N-l)\qi,t+l,opti) 

The single-agent probability is one when the policy of the option instructs the agent 
to communicate (i.e., TTopti{si,t) G T,), in which case the local process remains in the 
same local state. 

We use the notation s = (si,S2) and s' = (si,S2) refer to each agent's local state. 

Then, we denote by pf {s^,t-\-N\si,t,opti) the probability that agent i will reach state 
s'.i in N tim.e steps when it follows the option opti. It refers to the probability of 
reaching some state s\ without having terminated the option necessarily when this 
state is reached. This transition probability can be computed recursively since the 
transition probability of the underlying Dec-MDP is known: 

r Pi{s'^Si,Wopt,{si,t))) if7V=l 
pf{s'i,t+N\si,t,opti) = I otherwise 

I J2g^eS.Pis',\q^,^ToptM^,t))P^ {s'„ (t+l) + {N - l)\s,,t+l, Opt,) 

Finally, we obtain that: 

P^{s', t+N\s, t, opti,opt2) = Pl^{s[,t-\-N\si,t, opti) ■ P2 {S2,t+N\s2, t, opt2)+ 
Pi^{s'2,t+N\s2,t,opt2) - Pi {s[,t+N\sut,opti) 
-Pfis[,t+N\sut, opti) ■ Pi^{s'2, t-\-N\s2, t, opt2) 



Pl'{s'i,t+N\si,t,opti) = < 
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• R^{s, t, opti,opt2,s', t+N) is the expected reward obtained by the system N time steps 

after the agents started options opti and opt2 respectively in state s at time t, when 
at least one of them has terminated its option with a communication act (resulting in 
the termination of the other agent's option). This reward is computed for t+N < T. 

T^Nt , , , > ,^.n j C{opti,opt2,s,s',N) ift+N = T 

R {s, t, optuopt2, s , t+N) = I c{opt^,opt2, s, s', N) + otherwise 

C{opti, opt2, s, s' , N) is the expected cost incurred by the system when it transitions 
between states s and s' and at least one agent communicates after N time steps. We 
define the probability of a certain sequence of global states being transitioned by the 
system when each agent follows its corresponding option as P(<s°, s^ , . . . , s^ >): 

P(< 5\ . . . , 5^ >) = a n P{s^'\s\7ropt, (4), TTopt, (4)) 

j=o 

a is a normalizing factor that makes sure that over all possible sequences, the prob- 
ability adds up to one for a given s^, s^ and N steps going through intermedi- 
ate steps s^ , . . . , s^~^ . Then, we denote by Rseq the reward attained by the sys- 
tem when it traverses a certain sequence of states. Formally, Rseq{< s^,...,s^ > 
) = Z)^^ -^('S'') TTopti 7ropt2(s2), s-^^) where IT opti (si)) f^f^fs to the primitive action 
that is chosen by the option at the local state si- Finally, we can define the expected 
cost C{opti,opt2,s, s', N) as follows: 

C{opti ,opt2,s,s',N)= P{<s,q\..., q^-^ , s' >)Rseq{< s,q\...,q^-\s'>) 

Qi,...,g^-ies 

The dynamics of a semi-Markov decentralized process are as follows. Each agent per- 
forms its option starting in some global state s that is fully observed. Each agent's option is 
a mapping from local states to actions, so agent i starts the option in state Si at time t until 
it terminates in some state s'^, k time steps later. Whenever the options are terminated, 
the agents can fully observe the global state due to the terminal communication actions. If 
they reach state s' at time t+k<T, then the joint policy chooses a possible different pair 
of options at state s' at time t+k and the process continues. 

Communication in our model leads to a joint exchange of messages. Therefore all the 
agents observe the global state of the system once information is exchanged. This means that 
all those states of the decentralized semi-Markov process are fully- observable (as opposed 
to jointly fully-observable states as in the classical Dec-MDP-Com). 

The local policy for agent i in the Dec-SMDP-Com is a mapping from the global states 
to its options (as opposed to a mapping from sequences of observations as in the general 
Dec-POMDP case, or a mapping from a local state as in the Dec-MDPs with independent 
transitions and observations): 

5i : S X T ^ Opti 
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A joint policy is a tuple of local policies, one for each agent, i.e., a joint policy instructs 
each agent to choose an option in each global state. Thus, solving for an optimal mecha- 
nism is equivalent to solving optimally a decentralized semi-Markov decision problems with 
temporally abstracted actions. 

Lemma 1 A Dec-SMDP-Com is equivalent to a multi-agent MDP. 

Proof. Multiagent MDPs (MMDPs) represent a Markov decision process that is controlled 
by several agents (Boutilier, 1999). One important feature of this model is that all the 
agents have a central view of the global state. Formally, MMDPs are tuples of the form 
< Ag, {Ai}i(zAg, S, Pr, R >, where: 

• Ag is a finite collection of n agents. 

• {Ai}iQAg represents the joint action space. 

• iS is a finite set of system states. 

• Pr{s'\s,ai, . . . ,an) is the transition probability between global states s and s' when 
the agents perform a joint action. 

• R : S ^ ^ is the reward that the system obtains when a global state is reached. 

A decentralized semi-Markov problem with direct communication can be solved opti- 
mally by solving the corresponding MMDP. For simpicity of exposition we show the proof 
for systems with two agents. Following Definition 5, a 2-agent Dec-SMDP-Com is given 
by the tuple: < M , Opti, Opt2, P^ , R'^ >■ The mapping between these two models is as 
follows: Ag is the same finite collection of agents that control the MMDP and the semi- 
Markov process. The set S is the set of states of the world in both cases. In the MMDP 
model, these states are fully observable by definition. In the semi-Markov decentralized 
model these global states are also fully observable because agents always exchange informa- 
tion at the end of any option that they perform. The set of joint actions {Ai}i^Ag is given 
in the semi-Markov process as the set of options available to each agent (e.g., if n = 2 then 
{Ai} = {0pti,0pt2})- The difference is that the joint actions are chosen from primitive 
domain actions in the MMDP and the options are temporarily abstracted actions which 
terminate with a communication act. The probability transition and the reward functions 
can be easily mapped between the models by matching P^ with Pr and R^ with R. 

The solution to an MMDP (or Dec-SMDP-Com) problem is a strategy that assigns a 
joint action (or a set of options) to each global state. Solving an MMDP with actions 
given as options solves the semi-Markov problem. Solving a semi-Markov problem when 
the options are of length two, i.e., each option is composed of exactly one primitive action 
followed by a communication action that tells the agent to communicate its observation 
solves the corresponding MMDP problem. □ 

Solving a decentralized semi-Markov process with communication is P-complete because 
of Lemma 1 and the polynomial complexity of single agent MDPs (Papadimitriou &: Tsit- 
siklis, 1987). However, the input to this problem not only includes the states but also a 
double exponential number of domain actions for each agent. As explained in the next 
section, each option can be represented as a tree, where: 1) the depth of an option is lim- 
ited by the finite horizon T and 2) the branching factor of an option is constrained by the 
number of states in S. Therefore, the maximal number of leaves an option might have is 
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bounded by \S\'^ . Consequently, there can be assignments of primitive domain and 

communication acts to the leaves in each possible option. 

The naive solution to a Dec-SMDP-Com problem is to search the space of all possible 
pairs of options and find the pair that maximizes the value of each global state. The multi- 
step policy-iteration algorithm, presented in Section 5, implements a heuristic version of this 
search that converges to the optimal mechanism. The resulting search space (after pruning) 
can become intractable for even very simple and small problems. Therefore, we propose 
to apply communication-based decomposition mechanisms on restricted sets of options. 
Solving a Dec-SMDP-Com with a restricted set of options means to find the optimal policy 
that attains the maximal value over all possible options in the restricted set (Sutton et al., 
1999; Puterman, 1994). Sections 6 and 7 present two additional algorithms that solve 
Dec-SMDP-Com problems when the options considered are goal-oriented options, i.e., the 
mechanism assigns local goals to each one of the agents at each global state, allowing them 
to communicate before having reached their local goals. 

5. Multi-step Backup Policy- Iteration for Dec-SMDP-Com 

Solving a Dec-SMDP problem optimally means computing the optimal pair of options for 
each fully-observable global state. These options instruct the agents how to act indepen- 
dently of each other until information is exchanged. In order to find these options for each 
global state, we apply an adapted and extended version of the multi-step backup policy- 
iteration algorithm with heuristic search (Hansen, 1997). We show that the decentralized 
version of this algorithm converges to the optimal policy of the decentralized case with 
temporally abstracted actions. 

We extend the model of the single-agent POMDP with observations costs to the Dec- 
SMDP-Com model. From a global perspective, each agent that follows its own option 
without knowing the global state of the system, is following an open-loop policy. However, 
locally, each agent is following an option, which docs depend on the agent's local observa- 
tions. We first define a multi-step backup for options, when s and s' are global states of the 
decentralized problem: V{s, t, T) = 

max { ^ ^P'^{s',t+k\s,t,opti,opt2)[R^is,t,opti,opt2,s',t+k) + V{s',t+k,T)]} 

opti,opt2eOPTb 

OPT;, is the set of options of length at most b, where the length is defined as follows: 

Definition 6 (The lengtli of an Option) The length of an option is k if the option can 
perform at most k domain actions in one execution. 

As in Hansen's work, 6 is a bound on the length of the options {k <b). Here, the finite 

horizon Dec-SMDP-Com case is analyzed, therefore b <T. {s' ,t+k\s,t, opti, opt2) and 
{s,t,opti,opt2, s' ,t + k) are taken from the Dec-SMDP-Com model (Definition 5). 
We apply the multi-step backup policy-iteration algorithm (see Figure 2) using the 
pruning rule introduced by Hansen (1997), which we adapt to work on pairs of policies 
instead of linear sequences of actions. The resulting optimal multi-step backup policy is 
equivalent to the optimal policy of the MMDP (Lemma 1), i.e., it is equivalent to the optimal 
decentralized policy of a Dec-SMDP-Com with temporally abstracted actions. In order to 
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Figure 1: A policy tree of size k=3. 

explain the pruning rule for the decentralized case with temporally abstracted actions, we 
define what policy-tree structures are. 

Definition 7 (Policy-tree) A policy-tree is a tree structure, composed of local state nodes 
and corresponding action nodes at each level. Communication actions can only be assigned 
to leaves of the tree. The edges connecting an action a (taken at the parent state si) with a 
resulting state s'^ have the transition probability Pi{s[\si,a) assigned to them. 

Figure 1 shows a possible policy-tree. An option is represented by a policy-tree with 
all its leaves assigned communication actions. We denote a policy-tree Srootct, by the state 
assigned to its root (e.g., Sj-oot), and an assignment of domain actions and local states to 
the rest of the nodes (e.g., a). The size of a policy-tree is defined as follows: 

Definition 8 (Size of a Policy-tree) The size of a policy-tree is k if the longest branch 
of the tree, starting from its root is composed of k — 1 edges (counting the edges between 
actions and resulting states). A policy-tree of size one includes the root state and the action 
taken at that state. 

7r(ajt) is the policy induced by the assignment a with at most k actions in its implemen- 
tation. The expected cost g of a policy-tree Sj-ootc^k is the expected cost that will be incurred 
by an agent when it follows the policy iT{ak). We denote the set of nodes in a tree that do 
not correspond to leaves as NL and the set of states assigned to them Snl- The notation 
a\n refers to the a assignment excluding node n. The expected cost of a tree, g{srootOik)i 
is computed as follows: 



g{Srootak) 



C {aroot) 

C{aroot) + Es'g5a 



[Pr{s\\sroot, aroot)g{s'i{a \ root)k-i)\ 



iik = l 

if 1< A; < T 



Since the decentralized process has factored states, we can write a global state s as a 
pair (si, S2). Each agent can act independently of each other for some period of time k while 
it performs an option. Therefore, we can refer to the information state of the system after 
k time steps as siafcS2/3fc) where siak and S2/3fc correspond to each agent's policy tree of 
size k. We assume that at least one agent communicates at time t-\-k. This will necessarily 



180 



Communication-Based Decomposition Mechanism 



interrupt the other agent's option at the same time t+k. Therefore, it is sufficient to look 
at pairs of trees of the same size k. The information state refers to the belief an agent 
forms about the world based on the partial information available to it while it operates 
locally. In our model, agents get full observability once they communicate and exchange 
their observations. 

The heuristic function that will be used in the search for the optimal decentralized joint 
policy of the Dec-SMDP-Com follows the traditional notation, i.e., /(s) = g{s) + h{s). 
In our case, these functions will be defined over pairs of policy-trees, i.e., f{sakf3k) = 
G{sak/3k) + H{sakPk)- The / value denotes the backed-up value for implementing policies 
7r(afc) and Tr{f3k), respectively by the two agents, starting in state s at time t. The expected 
value of a state s at time t when the horizon is T is given by the multi-step backup for state 
s as follows: 

V{s,t,T)= max 

|aM/3|< b 

Note that the policy-trees corresponding to the assignments a and /3 are of size at most 
b < T. We define the expected cost of implementing a pair of policy-trees, G, as the sum 
of the expected costs of each one separately. If the leaves have communication actions, the 
cost of communication is taken into account in the g functions. As in Hansen's work, when 
the leaves are not assigned a communication action, we assume that the agents can sense 
at no cost to compute the / function. 

G{siai^2l3k) = 9{siak) + g{s2h)- 

An option is a policy-tree with communication actions assigned to all its leaves. That 
option is denoted by opti{ak) (or opt2{/3k))- The message associated with a leaf corresponds 
to the local state that is assigned to that leaf by a (or /3) . We define the expected value of 
perfect information of the information state saP after k time steps: 

H{sakPk) = {s ,t+k\s,t,opti{ak),opt2Wk))V{s' ,t+k,T) 

s'es 

The multi-step backup policy-iteration algorithm adapted from Hansen to the decen- 
tralized control case appears in Figure 2. Intuitively, the heuristic search over all possible 
options unfolds as follows: Each node in the search space is composed of two policy-trees, 
each representing a local policy for one agent. The search advances through nodes whose / 
value (considering both trees) is greater than the value of the global root state (composed 
of the roots of both policy-trees) . All nodes whose / value does not follow this inequality 
arc actually pruned and are not used for updating the joint policy. The policy is updated 
when a node, composed of two options is found for which f > V. All the leaves in these 
options (at all possible depths) include communication acts. The updated policy S' maps 
the global state s to these two options. When all the leaves in one policy-tree at current 
depth i have communication actions assigned, the algorithm assigns communication acts to 
all the leaves in the other policy-tree at this same depth. This change in the policies is cor- 
rect because there is joint exchange of information (i.e., all the actions are interrupted when 
at least one agent communicates). We notice, though, that there may be leaves in these 
policy-trees at depths lower than i that may still have domain actions assigned. Therefore, 
these policy-trees cannot be considered options yet and they remain in the stack. Any leaves 
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1. Initialization: Start with an initial joint policy 5 that assigns a pair of options to each global state s. 

2. Policy Evaluation: Vs G S,V^(s,t, T) = 

Y.lllj2s' P''{s\t + k\s,t,noptAsi,t),nopt2{s2,t))[R^{s,t,nopt^{si,t),nopt2{s2,t),s\t+k) + V\s\t+^^ 

3. Policy Improvement: For each state s = (si,S2) G S : 

a. Set-up: 

Create a search node for each possible pair of policy-trees with length 1: (siQi, S2/3i)- 
Compute /(sai/3i) = G(sai/3i) + H{sa\f5{). 
Push the search node onto a stack. 

b. While the search stack is not empty, do: 

i. Get the next pair of policy-trees: 

Pop a search node off the stack and let it be (siOj, S2/3i) 
(the policy-trees of length i starting in state s = (si, S2)) 
Let f{saiPi) be its estimated value. 

ii. Possibly update policy: 

if {f{soL,Pi) = G{saipi) + H{sa,f3,)) > V(s,t,T), then 

if all leaves at depth i in cither at or Pi have a communication action assigned, then 

Assign a communication action to all the leaves in the other policy-tree at depth i 
if all leaves in depths < i in both a; and Pi have a communication action assigned, then 

Denote these new two options opt\ and opt2- 

Let 5'{s) = {opt\,opti) and V{s,t,T) = f{saiPi). 

iii. Possibly expand node: 

If if{sa^Pi) = G{saiPi)+H{saiPi)) > V{s,t,T), then 

if ((some of the leaves in either a; or pi have domain actions assigned) and 
((i + 2) < T)) then 

/*At t + 1 the new action is taken and there is a transition to another state at t+2*/ 

Create the successor node of the two policy-trees of length i, 

by adding all possible transition states and actions to each leaf of each tree 

that does not have a communication action assigned to it. 

Calculate the / value for the new node (i.e., either f{sai+ipi+i) if both policy 
trees were expanded, and recalculate f{saiPi) if one of them has communication 
actions in all the leaves at depth i) 
Push the node onto the stack. 
/*A11 nodes with f <V are pruned and are not pushed to the stack.*/ 

4. Convergence test: 

lis = 6' then 

return S' 
else set 6 = 6', GOTO 2. 



Figure 2: Multi-step Backup Policy-iteration (MSBPI) using depth-first branch-and-bound. 



that remain assigned to domain actions will be expanded by the algorithm. This expansion 
requires the addition of all the possible next states, that are reachable by performing the 
domain-actions in the leaves, and the addition of a possible action for each such state. If all 
the leaves at depth i of one policy-tree are already assigned communication acts, then the 
algorithm expands only the leaves with domain actions at lower levels in both policy-trees. 
No leaf will be expanded beyond level i because at the corresponding time one agent is 
going to initiate communication and this option is going to be interrupted anyways. 

In the next section, we show the convergence of our Multi-step Backup Policy-iteration 
(MSBPI) algorithm to the optimal decentralized solution of the Dec-SMDP-Com, when 
agents follow temporally abstracted actions and the horizon is finite. 
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5.1 Optimal Decentralized Solution with Multi-step Backups 

In this section, we prove that the MSBPI algorithm presented in Figure 2 converges to 
the optimal dcccntrahzed control joint policy with temporally abstracted actions and direct 
communication. We first show that the policy improvement step in the algorithm based on 
heuristic multi-step backups improves the value of the current policy if it is sub-optimal. 
Finally, the policy iteration algorithm iterates over improving policies and it is known to 
converge. 

Theorem 1 When the current joint policy is not optimal, the policy improvement step in 
the multi-step backup policy-iteration algorithm always finds an improved joint policy. 

Proof. We adapt Hansen's proof to our decentralized control problem, when policies are 
represented by policy-trees. Algorithm MSBPI in Figure 2 updates the current policy when 
the new policy assigns a pair of options that yield a greater value for a certain global state. 
We show by induction on the size of the options, that at least for one state, a new option 
is found in the improvement step (step S.b.ii). 

If the value of any state can be improved by two policy-trees of size one, then an improved 
joint policy is found because all the policy-trees of size one are evaluated. We initialized S 
with such policy-trees. We assume that an improved joint policy can be found with policy- 
trees of size at most k. We show that an improved joint policy is found with policy-trees 
of size k. Lets assume that is a policy tree of size k, such that f{sak(3) > V{s) with 
communication actions assigned to its leaves. If this is the case then the policy followed 
by agent 2 will be interrupted at time k at the latest. One possibility is that sakP is 
evaluated by the algorithm. Then, an improved joint policy is indeed found. If this pair of 
policy-trees was not evaluated by the algorithm, it means that a was pruned earlier. We 
assume that this happened at level i. This means that f{saiP) < V{s). We assumed that 
f{sakl3) > V{s) so we obtain that: fisa^P) > f{sai/3). 

If we expand the / values in this inequality, we obtain the following: 

g{sai)+g{sl3)+Y,P''{s',tM\s,opti{ai),opt2{m9{s'aii,k))+g{s'l3)+Y,P''{s^^ > 

s' s" 

g{sai) + g{sp) + ^ {s' ,t-\-i\s, opti{ak), opi2(/3))F(s', t+i) 

where g{s'a{i, k)) refers to the expected cost of the subtree starting at level i and ending 
at level k starting from s'. After simplification we obtain: 

J2 P^'W, t+i\s, opii(ai), opt2(/3))[5(s'a(«, k)) + g{s' P) + ^ P^(s", t+i^k-i)V{s")] > 

s' s" 

P^{s', t+i\s, opti{au), opt2{P))V{s', t+i) 

s' 

That is, there exists some state s' for which f{s'a{i, k)j3) > V{s'). Since the policy-tree 
a{i,k) has size less than k, by the induction assumption we obtain that there exists some 
state s' for which the multi-step backed-up value is increased. Therefore, the policy found 
in step S.b.ii is indeed an improved policy. □ 



183 



Goldman & Zilberstein 



Lemma 2 The com,plexity of computing the optimal mechanism over general options by the 
MSBPI algorithm is 0(((|.4i| + |S|)(|A2| + (General options are based on any 

possible primitive domain action in the model, and any communication act). 

Proof. Each agent can perform any of the primitive domain actions in Ai and can com- 
municate any possible message in S. There can be at most leaves in a policy tree 
with horizon T and \S\ possible resulting states from each transition. Therefore, each time 
the MSBPI algorithm expands a policy tree (step S.b.iii in Figure 2), the number of result- 
ing trees is ((I All + |S|)(|A2| + 1^1))'""^'*^ In the worst case, this is the number of trees 
that the algorithm will develop in one iteration. Therefore, the size of the search space is a 
function of this number times the number of iterations until convergence. □ 

Solving optimally a Dec-MDP-Com with independent transitions and observations has 
been shown to be in NP (Goldman & Zilberstein, 2004a). As we show here, solving for the 
optimal mechanism is harder, although the solution may not be the optimal. This is due 
to the main difference between these two problems. In the Dec-MDP-Com, we know that 
since the transitions and observations are independent, a local state is a sufficient statistic 
for the history of observations. However, in order to compute an optimal mechanism we 
need to search in the space of options, that is, no single local state is a sufficient statistic. 
When options are allowed to be general, the search space is larger since each possible 
option that needs to be considered can be arbitrarily large (with the length of each branch 
bounded by T). For example, in the Meeting under Uncertainty scenario (presented in 
Section 7.1), agents aim at meeting in a stochastic environment in the shortest time as 
possible. Each agent can choose to perform anyone of six primitive actions (four move 
actions, one stay action and a communication action). Even in a small world composed of 
100 possible locations, implementing the MSBPI algorithm is intractable. It will require 
the expansion of all the possible combinations of pairs of policy-trees leading to a possible 
addition of 36^''°^^ nodes to the search space at each iteration. Restricting the mechanism 
to a certain set of possible options, for example goal-oriented options leads to a significant 
reduction in the complexity of the algorithm as we shown in the following two sections. 

6. Dec-SMDP-Com with Local Goal-oriented Behavior 

The previous section provided an algorithm that computes the optimal mechanism, search- 
ing over all possible combinations of domain and communication actions for each agent. 
On the one hand, this solution is the most general and does not restrict the individual 
behaviors in any aspect. On the other hand, this solution may require the search of a very 
large space, even after this space is pruned by the heuristic search technique. Therefore, in 
order to provide a practical decomposition mechanism algorithm, it is reasonable to restrict 
the mechanism to certain sets of individual behaviors. In this section, we concentrate on 
goal-oriented options and propose an algorithm that computes the optimal mechanism with 
respect to this set of options: i.e., the algorithm finds a mapping from each global state to 
a set of locally goal oriented behaviors with the highest value. The algorithm proposed has 
the same structure as the MSBPI algorithm; the main difference is in how the options are 
built. 
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Definition 9 (Goal-oriented Options) A goal- oriented option is a local policy that achieves 
a given local goal. 

We study locally goal-oriented mechanisms, which map each global state to a pair of 
goal-oriented options and a period of time k. We assume here that a set of local goals Gi 
is provided with the problem. For each such local goal, a local policy that can achieve it is 
considered a goal-oricntcd option. When the mechanism is applied, each agent follows its 
policy to the corresponding local goal for k time steps. At time k + 1, the agents exchange 
information and stop acting (even though they may not have reached their local goal) . The 
agents, then, become synchronized and they are assigned possibly different local goals and 
a working period k' . 

The algorithm presented in this section, LGO-MSBPI, solves the decentralized control 
problem with communication by finding the optimal mapping between global states to local 
goals and periods of time (the algorithm finds the best solution it can given that the agents 
will act individually for some periods of time). We start with an arbitrary joint policy that 
assigns one pair of local goal states and a number k to each global state. The current joint 
policy is evaluated and set as the cTirrcnt best known mechanism. Given a joint policy 
5 : S X T ^ Gi X G2 X T, {Gi C 5^ C 5), the value of a state s at a time t, when T is 
the finite horizon is given in Equation 1: (this value is only computed for states in which 
t+k < T). 

( i{t = T 

V\s,t,T) = I E.'es^j^(«^*+fc|«'^'^^l(«l)'^^2(s2))[<(s,^,7rg,,7rg,,s^fc) + T/^(5^^+fc,^)](l) 
[ s.t. S{s,t) = (51,52, A;) 

Notice that {s, Hg-^ jg-^ , s', k) can be defined similarly to (see Definition 5), taking 
into account that the options here are aimed at reaching a certain local goal state (tt^^ and 
TTpj are aimed at reaching the local goal states gi and §2, respectively). 

Rg{s,t,TTg~^,Trg'^,s',k) =C{ng-^,'Kg'^,s,s',k)+Cj: = 
Ce+ Y1 P{<s,q\...,q''-\s'>)-Cseg{<s,q\...,s''-\s'> 

There is a one-to-one mapping between goals and goal-oriented options. That is, the 
policy TTg. assigned by 5 can be found by each agent independently by solving optimally 
each agent's local process MDPi = (Si, Pi, Ri,Gi,T): The set of global states S is factored 
so each agent has its own set of local states. The process has independent transitions, so 
Pi is the primitive transition probability known when we described the options framework. 
Ri is the cost incurred by an agent when it performs a primitive action and zero if the 
agent reaches a goal state in Gi. T is the finite horizon of the global problem. 

P^ (with the goal g subscript) is different from the probability function P^ that appears 
in Section 4. P^ is the probability of reaching a global state s' after k time steps, while 
trying to reach gi and §2 respectively following the corresponding optimal local policies. 

P^{s', t + k\s, t + i, TTgi (si), 7rg2(s2)) = 
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1 




a i = k and s = s' 
if i = k and s ^ s' 
if i < fc 



< 



Es*es^(s*|s,7rg,(si),7rg,(s2)) 




(S', t+k\s*,t + i+l, TTg, {SD, TTg'^isD) 

s.t. 5{s,t+i) = (ffi,52,fc) 



Each iteration of the LGO-MSBPI algorithm (shown in Figure 3) tries to improve the 
value of each state by testing all the possible pairs of local goal states with increasing number 
of time steps allowed until communication. The value of / is computed for each mapping 
from states to assignments of local goals and periods of time. The / function for a given 
global state, current time, pair of local goals and a given period of time k expresses the cost 
incurred by the agents after having acted for k time steps and having communicated at time 
k+1, and the expected value of the reachable states after k time steps (these states are those 
reached by the agents while following their corresponding optimal local policies towards gi 
and g2 respectively). The current joint policy is updated when the / value for some state 
s, time t, local goals gi and 52 and period k is greater than the value V^{s,t,T) computed 
for the current best known assignment of local goals and period of time. Formally: 



C(7rgj , TTgj, s, fc) is the expected cost incurred by the agents when following the cor- 
responding options for k time steps starting from state s. This is defined similarly to the 
expected cost explained in Definition 5. We notice that the computation of / refers to the 
goals being evaluated by the algorithm, while the evaluation of the policy (step 2) refers to 
the goals assigned by the current best policy. 

6.1 Convergence of the Algorithm and Its Complexity 

Lemma 3 The algorithm LGO-MSBPI in Figure 3 converges to the optimal solution. 

Proof. The set of global states S and the set of local goal states Gi C S are finite. 
The horizon T is also finite. Therefore, step 3 in the algorithm will terminate. Like the 

classical policy-iteration algorithm, the LGO-MSBPI algorithm also will converge after a 
finite numbers of calls to step 3 where the policy can only improve its value from one 
iteration to another. □ 

Lemma 4 The complexity of computing the optimal mechanism based on local goal-oriented 
behavior following the LGO-MSBPI algorithm is polynomial in the size of the state space. 

Proof. Step 2 of the LGO-MSBPI algorithm can be computed with dynamic programming 
in polynomial time (the value of a state is computed in a backwards manner from a finite 
horizon T). The complexity of improving a policy in Step 3 is polynomial in the time. 



f{s,t,gi,g2,k) 
G{s,t,gi,g2,k) 



G{s,t,gi,g2,k) H{s,t, gi, g2, k) 

C{T^g\,T^g2,S,t,k) + Cs 

' \it = T 

iit<T 

Es'eS Pg^'is', t + k\s, t, -Kg, (Sl), TTg^ {S2))V^ {s' , t + k,T) 



(2) 
(3) 



H{s,t,gi,g2,k) 



(4) 
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1. Initialization: Start with an initial joint policy S that assigns local goals 

Qi G Gi and time periods k Cz N 
yseS,t: S{s,t)^{g,,g2,k) 

2. Policy Evaluation: Vs e S, Compute V^{s,t,T) based on Equation 1. 

3. Policy Improvement: 

a. fc = 1 

b. While (k < T) do 

i. Vs,t, 31,(72: Compute fis,t,gi,g2,k) based on Equations 2,3 and 4. 

ii. Possible update policy 

if f{s,t,gi,g2,k) > V^{s,t,T) then 

S{s,t) <— (gi, 92, k) /* Communicate at k + 1*/ 
V'{s,t,T)^'f{s,t,guh,k) 

iii. Test joint policies for next extended period of time 

k^k + \ 

4. Convergence test: 

if 5 did not change in Step 3 then 

return 6 
else GOTO 2. 



Figure 3: Multi-step Backup Policy-iteration with local goal-oriented behavior 
(LGO-MSBPI). 



number of states and number of goal states, i.e., (9(T^|S'||(_r|). In the worst case, every 
component of a global state can be a local goal state. However, in other cases, \Gi\ can be 
much smaller than l^il when Gi is a strict subset of Si, decreasing even more the running 
time of the algorithm. □ 

6.2 Experiments - Goal-oriented Options 

We illustrate the LGO-MSBPI decomposition mechanism in a production control scenario. 

We assume that there are two machines, which can control the production of boxes and 
cereals: machine Mi can produce two types of boxes a or h. The amoTint of boxes of type 
a produced by this machine is denoted by Ba (Bf, represents the amount of boxes of type b 
produced respectively). Machine M2 can produce two kinds of cereals a and b. Ca (and C}, 
respectively) denotes the number of bags of cereals of type a (we assume that one bag of 
cereals is sold in one box of the same type). The boxes differ in their presentation so that 
boxes of type a advertise their content of type a and boxes of type 6 advertise their content 
of type b. We assume that at each discrete time t, machine M\ may produce one box or 
no boxes at all, and the other machine may produce one bag of cereals or may produce 
no cereal at all. This production process is stochastic in the sense that the machines are 
not perfect: with probability -Pmi, machine one succeeds in producing the intended box 
(either a or b) and with probability 1 — Pmi , the machine does not produce any box in that 
particular time unit. Similarly, we assume expresses the probability of machine two 
producing one bag of cereals of type a or 6 that is required for selling in one box. In this 
example, the reward attained by the system at T is equal to the number of products ready 
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for sale, i.e., min{Ba, Ca} + min{Bh, C^}. A product that can be sold is composed of one 
box together with one bag of cereals corresponding to the type advertised in this box. 

A goal-oriented option is given by the number of products that each machine should 
produce. Therefore, an option opU in this scenario is described by a pair of numbers {Xa, Xb) 
(when i is machine one then X refers to boxes and when i is machine two, X refers to bags 
of cereals). That is, machine i is instructed to produce Xa items of type a, followed by 
Xf) items of type b, followed by Xa items of type a and so forth until either the time limit 
is over or anyone of the machines decides to communicate. Once the machines exchange 
information, the global state is revealed, i.e., the current number of boxes and cereals 
produced so far is known. Given a set of goal-oriented options, the LGO-MSBPI algorithm 
returned the optimal joint policy of action and communication that solves this problem. 
We counted the time units that it takes to produce the boxes with cereals. We compared 
the locally goal oriented multi-step backup policy iteration algorithm (LGO-MSBPI) with 
two other approaches: 1) the Ideal case when machines can exchange information about 
their state of production at each time and at no cost. This is an idealized case, since 
in reality exchanging information does incur some cost, for example changing the setting 
of a machine takes valuable time and 2) the Always Communicate ad-hoc case, when the 
machines exchange information at each time step and they also incur a cost when they 
do it. Tables 1, 2, and 3 present the average utility obtained by the production system 
when the cost of communication was set to —0.1, —1 and —10 respectively, the cost of a 
domain action was set to —1 and the joint utility was averaged over 1000 experiments. A 
state is represented by the tuple (Ba, Bii,Ca,Cb)- The initial state was set to (0,0,0,8), 
there were no boxes produced and there were already 8 bags of cereals of type B. The 
finite horizon T was set to 10. The set of goal-oriented options {Xa,Xh) tested included 
(0,1),(1,4),(2,3),(1,1),(3,2),(4,1) and (1,0). 





Av{^rag{^ Utility 


I(k>al C'i: = 


Always Communicate 


LGO-ALSBPI 


0.2,0.2 


-17.012 


-18.017 


-17.7949 


0.2,0.8 


-16.999 


-17.94 


-18.0026 


0.8,0.8 


-11.003 


-12.01 


-12.446 


Table 1: Cs = -0.10, i?„ = -1.0. 


Pmi , 


Average Utility 


Ideal Ce = 


Always Communicate 


LGO-MSBPI 


0.2,0.2 


-17.012 


-26.99 


-19.584 


0.2,0.8 


-16.999 


-26.985 


-25.294 


0.8,0.8 


-11.003 


-20.995 


-17.908 



Table 2: Cs = -1.0, Ra = -1.0. 



The LGO-MSBPI algorithm computed a mechanism that resulted in three products on 
average when the uncertainty of at least one machine was set to 0.2 and 1000 tests were 
run, each for ten time units. The number of products increased on average between 8 to 
9 products when the machines succeeded 80% of the cases. These numbers of products 
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Pmi , 


Average Utility 


Ideal Ce = 


Always Communicate 


LGO-MSBPI 


0.2,0.2 


-17.012 


-117 


-17.262 


0.2,0.8 


-16.999 


-117.028 


-87.27 


0.8,0.8 


-11.003 


-110.961 


-81.798 



Table 3: Ce = -10.0, Ra = -1.0. 



were always attained either when the decomposition mechanism was implemented or when 
the ad- hoc approaches were tested. Ideal or Always Communicate algorithms only differ 
with respect to the cost of communication, and they do not differ in the actual policies of 
action. Although the machines incur a higher cost when the mechanism is applied compared 
to the ideal case (due to the cost of communication), the number of final products ready 
to sell were almost the same amount. That is, it will take some more time in order to 
produce the right amount of products when the policies implemented are those computed 
by the locally goal oriented multi-step backup policy iteration algorithm. The cost of 
communication in this scenario can capture the cost of changing the setting of one machine 
from one production program to another. Therefore, our result is significant when this cost 
of communication is very high compared to the time that the whole process takes. The 
decomposition mechanism finds what times are most beneficial to synchronize information 
when constant communication is not feasible nor desirable due to its high cost. 

6.3 Generalization of the LGO-MSBPI Algorithm 

The mechanism approach assumes that agents can operate independent of each other for 
some period of time. However, if the decentralized process has some kind of dependency 
in its observations or transitions, this assumption will be violated, i.e., the plans to reach 
the local goals can interfere with each other (the local goals may not be compatible). The 
LGO-MSBPI algorithm presented in this paper can be applied to Dec-MDPs when their 
transitions and observations are not assumed to be independent. In this section, we bound 
the error in the utilities of the options computed by the LGO-MSBPI algorithms when such 
dependencies do exist. We define A— independent decentralized processes to refer to nearly- 
independent processes whose dependency can be quantified by the cost of their marginal 
interactions. 

Definition 10 {A— independent Process) Let C^. (s — >■ gkldj) be the expected cost in- 
curred by agent i when following its optimal local policy to reach local goal state gt from 
state s, while the other agent is following its optimal policy to reach gj. A decentralized 
control process is A — independent if A = max{Ai, A2}, where Ai and A2 are defined as 
follows: ygi,g'i e Gi G ^i, ^2,52 & G2 & S2 and s G 5; 

Ai = max{max{max{CAi (s° ^ 51I52) - Cai{s° 9i\92)}}} 

* 91 92,92 

A2 = max{max{max{CA2 (s° ^ g2\g'i) - Ca2{s° hlgi)}}} 

* 92 gi,g[ 



189 



Goldman & Zilberstein 



That is, A is the maximal difference in cost that an agent may incur when trying to 
reach one local goal state that interferes with any other possible local goal being reached by 
the other agent. 

The computation of the cost function C^. (s — t- gk\gj) is domain dependent. We do not 
address the issue of how to compute this cost but we provide the condition. The individual 
costs of one agent can be affected by the interference that exists between some pair of local 
goals. For example, assume a 2D grid scenario: one agent can move only in four directions 
(north, south, east and west) and needs to reach location (9,9) from (0,0). The second agent 
is able of moving and also of collecting rocks and blocking squares in the grid. Assuming 
that the second agent is assigned the task of blocking all the squares in even rows, then the 
first agent's solution to its task is constrained by the squares that are free to cross. In this 
case, agent one's cost to reach (9,9) depends on the path it will choose that depends very 
strongly on the state of the grid resulting from the second agent's actions. 

The A value denotes the amount of interference that might occur between the agents' 
locally goal-oriented behaviors. When the Dec-MDP has independent transitions and ob- 
servations, the value of A is zero. The LGO-MSBPI algorithm proposed in this paper 
computes the mechanism for each global state as a mapping from states to pairs of local 
goal states ignoring the potential interference. Therefore, the difference between the actual 
cost that will be incurred by the options found by the algorithm and the optimal options 
can be at most A. Since the mechanism is applied for each global state for T time steps 
and this loss in cost can occur in the worst case for both agents, the algorithm presented 
here is 2T A — optimal in the general case. 

7. A Myopic-greedy Approach to Direct Communication 

In some cases, it is reasonable to assume that single- agent behaviors are already known 
and fixed, ahead of time for any possible global state. For example, this may occur in 
settings where individual agents are designed ahead of the coordination time (e.g., agents 
in a manufacturing line can represent machines, which are built specifically to implement 
certain procedures). To achieve coordination, though, some additional method may be 
needed to synchronize these individual behaviors. In this section, we present how to apply 
the communication-based decomposition approach to compute the policy of communication 
that will synchronize the given goal-oriented options. We take a myopic-greedy approach 
that runs in polynomial-time: i.e., each time an agent makes a decision, it chooses the 
action with maximal expected accumulated reward assuming that agents are only able 
to communicate once along the whole process. Notice that the LGO-MSBPI was more 
general in the sense that it also computed what local goals should be pursued by each 
agent together with the communication policy that synchronizes their individual behaviors. 
Here, each time the agents exchange information, the mechanism is applied inducing two 
individual behaviors (chosen from the given mapping from states to individual behaviors). 
The given optimal policies of action (with no communication actions) are denoted (5^* and 
$2* respectively. 

The expected global reward of the system, given that the agents do not communicate 
at all and each follows its corresponding optimal policy 6f* is given by the value of the 
initial state s°: e^c(s°> -^i^*, ^2^*)- This value can be computed by summing over all possible 
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next states and computing the probability of each agent reaching it, the reward obtained 
then and the recursive value computed for the next states. 

(s'j,s!,) 

We denote the expected cost of the system computed by agent i, when the last synchronized 
state is s^, and when the agents communicate once at state s and continue without 
any communication, Bc(s°, Sj, (5]^*, 5^*): 

E P^{s2\sl Stms", S^*{s<i), St{s°2), {suS2)) + eiJis,, s2),S^*,5t) + ■ Flag) 

Flag is zero if the agents reached the global goal state before they reached state s. The 
time stamp in state s is denoted t{s). P{s\, iJ^^*, ^2^*) is the probability of reaching state 
s from state following the given policies of action. 

{1 if s = s' 

P{s'\s, 5f 6tis2)) if t{.s') = tis) + 1 

if t{s') < t{s) + 1 
T,s"esPis'\s" ■ Pis"\s,6^*,St) otherwise 

Similarly, Pi (P2) can be defined for the probability of reaching s[ (S2), given agent 1 (2)'s 
current partial view si (§2) and its policy of action Si^* {62*)- The accumulated reward 
attained while the agents move from state s° to state s is given as follows: 



R{s'',S^*,6t,s) 



{ R{s\ Sf*{s% 6^*{s^2), s) if t{s) = + 1 

if t{s) > t{s°) + 1 

E." W I s^* , St ,s°)-p{s\6t, Si* ,s"y 

[ {R{s°, St, St,s") + R{s",St{sl), St {4), s)) 



At each state, each agent decides whether to communicate its partial view or not based on 

whether the expected cost from following the policies of action, and having communicated 
is larger or smaller than the expected cost from following these policies of action and not 
having communicated. 

Lemma 5 Deciding a Dec-MDP-Com with the myopic- greedy approach to direct commu- 
nication is in the P class. 

Proof. Each agent executes its known policy Sf* when the mechanism is applied. If local 
goals are provided instead of actual policies, finding the optimal single-agent policies that 
reach those goal states can be done in polynomial time. The complexity of finding the 
communication policy is the same as dynamic programming (based on the formulas above), 
therefore computing the policy of communication is also in P. There are \ S\ states for which 
and Qc need to be computed, and each one of these formulas can be solved in time 
polynomial in ISI. □ 
In previous work, we have also studied the set of monotonic goal-oriented Dec-MDPs, 
for which we provide an algorithm that finds the optimal policy of communication assuming 
a set of individual behaviors is provided (Goldman & Zilberstein, 2004b). 
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7.1 Meeting Under Uncertainty Example 

We present empirical results obtained when the myopic-greedy approach was applied to the 
Meeting under Uncertainty example^. The tcstbcd we consider is a sample problem of a Dec- 
MDP-Com involving two agents that have to meet at some location as early as possible. This 
scenario is also known as the gathering problem in robotics (Suzuki &: Yamashita, 1999). 
The environment is represented by a 2D grid with discrete locations. In this example, any 
global state that can be occupied by both agents is considered a global goal state. The set 
of control actions includes moving North, South, East and West, and staying at the same 
location. Each agent's partial view (which is locally fully-observable) corresponds to the 
agent's location coordinates. The observations and the transitions are independent. The 
outcomes of the agents' actions are uncertain: that is, with probability Pi, agent i arrives at 
the desired location after having taken a move action, but with probability I — Pi the agent 
remains at the same location. Due to this uncertainty in the effects of the agents' actions, 
it is not clear that setting a predetermined meeting point is the best strategy for designing 
these agents. Agents may be able to meet faster if they change their meeting place after 
realizing their actual locations. This can be achieved by exchanging information on the 
locations of the agents, that otherwise are not observable. We showed that exchanging the 
last observation guarantees optimality in a Dec-MDP-Com process with constant message 
costs (Goldman & Zilberstein, 2004a). In the example tested, the messages exchanged 
correspond to the agents' own observations, i.e., their location coordinates. 

We have implemented the locally goal-oriented mechanism that assigns a single local 
goal to each agent at each synchronized state. These local goals were chosen as the location 
in the middle of the shortest Manhattan path between the agents' locations (this distance 
is revealed when information is exchanged). 

Intuitively, it is desirable for a mechanism to set a meeting place in the middle of the 
shortest Manhattan path that connects the two agents because in the absence of commu- 
nication, the cost to meet at that point is minimal. This can be shown by computing the 
expected time to meet, Qnc, for any pair of possible distances between the two agents and 
any location in the grid, when no communication is possible. To simplify the exposition, 
we use a function that takes advantage of the specific characteristics of the example. The 
notation is as follows: agent 1 is at distance di from the meeting location, agent 2 is at 
distance ^2 from that location, the system incurs a cost of one at each time period if the 
agents have not met yet. If both agents are at the meeting location, the expected time to 
meet is zero, Onc(0, 0) = 0. If only agent 2 is at the meeting location, but agent 1 has not 
reached that location yet, then the expected time to meet is given by 

e„e(di, 0) = Pi • (-1 + e„e(di-i, 0)) + (1-Pi) • (-1 + e„e(rfi, 0)) = 

= Pi • e„e(dl-l,0)) + (1-Pi) • G„e(rfl,0)) - 1 

That is, with probability Pi agent 1 succeeds in decreasing its distance to the meeting lo- 
cation by one, and with probability 1 — Pi it fails and remains at the same location. Recur- 
sively, we can compute the remaining expected time to meet with the updated parameters. 
Similarly for agent 2: Gnc(0, ^2) = P2 ■ (-1 + e„,(0, ^2-1)) + (I-P2) • (-l + enc(0, ^2)). If 
none of the agents has reached the meeting place yet, then there are four different cases in 
which either both, only one, or none succeeded in moving in the right direction and either 

4. Some of the empirical results in this section were described first by Goldman and Zilberstein (2003). 
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or not decreased their distances to the meeting location respectively: 

e„c(rfi,rf2) = Pi-P2- (-1 + e„c(rfi-i, d2-i)) + Pi ■ (1-P2) • (-1 + e„e(rfi-i,rf2))+ 

+ (1-Pl) • P2 • (-1 + e„e(dl,rf2-l)) + (l-^^l) • (I-P2) • (-1 + Onc{dl,d2)) = 

= Pi-P2- e„e(rfi- 1,^2-1) + Pi ■ (1-P2) • e„,(rfi-i,a!2) + (1-Pi) • P2 ■ e„e(di,d2-i)+ 

+(i-Pi)-(i-P2)-e„c(rfi,d2)-i 

The value of @nc{di, d2) was computed for all possible distances di and d2 in a 2D grid of 
size 10 X 10. The minimal expected time to meet was obtained when di = ^2 = 9 and the 
expected cost was —12.16. 

In summary, approximating the optimal solution to the Meeting under Uncertainty 
example when direct communication is possible and the mechanism applied is the one de- 
scribed above will unfold as follows: At time to, the initial state of the system s° is fully 
observable by both agents. The agents set a meeting point in the middle of a Manhat- 
tan path that connects them. Denote by do the distance between the agents at to and 
9to = {dial 9 to) S°^^ state set at Iq. Each one of the agents can move optimally towards 
its corresponding component of gt^ ■ Each agent moves independently in the environment 
because the transitions and observations arc independent. Each time t, when the policy of 
communication instructs an agent to initiate exchange of information, the current Manhat- 
tan distance between the agents df is revealed to both. Then, the mechanism is applied, 
setting a possibly new goal state gt, which decomposes into two components one for each 
agent. This goal state gt is in the middle of the Manhattan path that connects the agents 
with length dt revealed through communication. 

7.2 Experiments - Myopic-greedy Approach 

In the following experiments, we assumed that the transition probabilities Pi and P2 are 
equal. These uncertainties were specified by the parameter P„. The mechanism that is 
applied whenever the agents communicate at time t results in each agent adopting a local 
goal state, that is set at the location in the middle of the Manhattan path connecting the 
agents (the Manhattan distance between the agents is revealed at time t) . We compare the 
joint utility attained by the system in the following four different scenarios: 

1. No- Communication — The meeting point is fixed at time to and remains fixed along 
the simulation. It is located in the middle of the Manhattan path that connects 
between the agents, known at time to- Each agent follows its optimal policy of action 
to this location without communicating. 

2. Ideal — This case assumes that the agents can communicate freely (Cs = 0) at every 
time step resulting in the highest global utility that both agents can attain. Notice, 
though, that this is not the optimal solution wc are looking for, because we do assume 
that communication is not free. Nevertheless, the difference in the utility obtained in 
these first two cases shed light on the trade-off that can be achieved by implementing 
non-free communication policies. 

3. Communicate SubGoals — A heuristic solution to the problem, which assumes that 
the agents have a notion of sub-goals. They notify each other when these sub-goals 
are achieved, eventually leading the agents to meet. 
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4. Myopic-greedy Approach — Agents act myopically optimizing the choice of when to 
send a message, assuming no additional communication is possible. For each possible 
distance between the agents, a policy of communication is computed such that it 
stipulates the best time to send that message. By iterating on this policy agents are 
able to communicate more than once and thus approximate the optimal solution to 
the decentralized control problem with direct communication. The agents continue 
moving until they meet. 

The solution to the No-Communication case can be solved analytically for the Meeting 
under Uncertainty example, by computing the expected cost Qncidi,d2) incurred by two 
agents located at distances di and d2 respectively from the goal state at time to (the complete 
mathematical solution appears in Section 7.1). In the Ideal case, a set of 1000 experiments 
was run with cost of communication set to zero. Agents communicate their locations at 
every time instance, and update the location of the meeting place accordingly. Agents move 
optimally to the last synchronized meeting location. 

For the third case tested (Communicate SubGoals) a sub-goal was defined by the cells 
of the grid with distance equal to p ■ d/2 and with center located at d/2 from each one of 
the agents, p is a parameter of the problem that determines the radius of the circle that 
will be considered a sub-goal. Each time an agent reaches a cell inside the area defined 
as a sub-goal, it initiates exchange of information (therefore, p induces the communication 
strategy), d expresses the Manhattan distance between the two agents, this value is accurate 
only when the agents synchronize their knowledge. That is, at time to the agents determine 
the first sub-goal as the area bounded by a radius of p • do/2 and, which center is located at 
do/2 from each one of the agents. Each time t that the agents synchronize their information 
through communication, a new sub-goal is determined at p ■ dt/2. Figure 4 shows how 
new sub-goals are set when the agents transmit their actual location once they reached a 
sub-goal area. The meeting point is dynamically set at the center of the sub-goal area. 

Experiments were run for the Communicate SubGoals case for different uncertainty val- 
ues, values of the parameter p and costs of communication (for each case, 1000 experiments 
were run and averaged). These results show that agents can obtain higher utility by adjust- 
ing the meeting point dynamically rather than having set one fixed meeting point. Agents 
can synchronize their knowledge and thus they can set a new meeting location instead of 
acting as two independent MDPs that do not communicate and move towards a fixed meet- 
ing point (see Figure 5). Nevertheless, for certain values of p, the joint utility of the agents 



194 



Communication-Based Decomposition Mechanism 




is actually smaller than the joint utility achieved in the No- Communication case (2 MDPs 
in the figure). This points out the need to empirically tune up the parameters required by 

the heuristic. 

In the Myopic- greedy case, we design the agents to optimize the time when they should 
send a message, assuming that they can communicate only once. At the off-line planning 
stage, the agents compute their expected joint cost to meet for any possible state of the 
system and time t (included in the local state Si), 0c(s°, Si, S^*,52*)- The global states 
revealed through communication correspond to the possible distances between the agents. 
Each time the agents get synchronized, the mechanism is applied assigning local goals and 
instructing the agents to follow the optimal local policies to achieve them. In the Meeting 
under Uncertainty scenario we study, 6c is the expected joint cost incurred by taking control 
actions during t time steps, communicating then at time t -|- 1 if the agents have not met 
so far, and continuing with the optimal policy of control actions without communicating 
towards the goal state (the meeting location agreed upon at f -|- 1) at an expected cost of 
Qnc{di,d2) as computed for the No-Communication case. When the agents meet before the 
t time steps have elapsed, they only incur a cost for the time they act before they met. 

At each time t, each one of the agents knows a meeting location, that is the goal location 
computed from the last exchange of information. Consequently, each agent moves optimally 
towards this goal state. In addition, the myopic-greedy policy of communication is found 
by computing the earliest time t, for which @cidi + d2,si,6i-* ,82*) < Qnc{di,d2), that is, 
what is the best time to communicate such that the expected cost to meet is the least. The 
myopic-greedy policy of communication is a vector that states the time to communicate for 
each possible distance between the agents. 

Tables 4, 5, and 6 present the myopic-greedy communication policies computed for the 
Meeting under Uncertainty problem with P„ values taken from {0.2, 0.4, 0.6, 0.8}. The cost 
of taking a control action is Ra = —1.0 and the costs of communicating Cs tested were 
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{—0.1, —1.0, —10.0}. Each row corresponds to a configuration tested with different state- 
transition uncertainties. Each column corresponds to a synchronized state, given by the 
possible Manhattan distance between the agents moving in a 2D grid of size 10x10. Given 
a certain value for and a certain global distance, each agent interprets the value in 
any entry as the next time to communicate its position. Time is reset to zero when the 
agents exchange information. As long as the distance between the agents is larger and the 
communication cost increases, the policy instructs the agents to communicate later, i.e., 
the agents should keep operating until the information exchanged will have a better effect 
on the rescheduling of the meeting place. 
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Table 4: Myopic-greedy policy of communication: Cs = —0.1, Ra = —1.0. 
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Table 5: Myopic-greedy policy of communication: Cs = —1.0, Ra = —1.0. 
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Table 6: Myopic-greedy policy of communication: Cs = —10.0, Ra = —1.0. 



For the smallest cost of communication tested, it is always beneficial to communicate 
rather early, no matter the uncertainty in the environment, and almost no matter what 
do is. For larger costs of communication and a given P„, the larger the distance between 
the agents, the later they will communicate (e.g., when P„ = 0.4, Cs = —1 and d = 5, 
agents should communicate at time 4, but if Cs = —10, they should communicate at time 
9). For a given C^, the larger the distance between the agents is, the later the agents will 
communicate (e.g., when P„ = 0.4, Cs = —10 and d = 5, agents should communicate at 
time 9, but if d = 12, they should communicate at time 16). The results from averaging 
over 1000 runs show that for a given cost Cs as long as P„ decreases (the agent is more 
uncertain about its actions' outcomes), the agents communicate more times. 
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In the 1000 experiments run, the agents exchange information about their actual loca- 
tions at the best time that was myopically found for do (known to both at time to). After 
they communicate, they know the actual distance dt, between them. The agents follow the 
same myopic-greedy communication policy to find the next time when they should commu- 
nicate if they did not meet already. This time is the best time found by the myopic-greedy 
algorithm given that the distance between the agents was dt. Iteratively, the agents approx- 
imate the optimal solution to the decentralized control problem with direct communication 
by following their independent optimal policies of action, and the myopic-greedy policy 
for communication. Results obtained from averaging the global utility attained after 1000 
experiments show that these myopic-greedy agents can perform better than agents who com- 
municate sub-goals (that was shown already to be more efficient than not communicating 
at all). 



Pu 


Average Joint Utility 


No- Comm. 


Ideal Ce = 


SubCoals'^ 


Myopic- Greedy 


0.2 


-104.925 


-62.872 


-64.7399 


-63.76 


0.4 


-51.4522 


-37.33 


-38.172 


-37.338 


0.6 


-33.4955 


-26.444 


-27.232 


-26.666 


0.8 


-24.3202 


-20.584 


-20.852 


-20.704 



Table 7: Cs = -0.10, Ra = -1.0. 



The Myopic-greedy approach attained utilities statistically significantly greater than 
those obtained by the heuristic case when Cs = -0.1 (see Table 7)^. When Cs = -0.1 and 
Pu = 0.4, Myopic-greedy even attained utilities not significantly different (with significance 
level 98%) than Ideal. 



Pu 


Average Joint Utility 


No-Comm. 


Ideal Ce = 


Comm. SubGoals - Best p 


Myopic-greedy 


0.2 


-104.925 


-62.872 


-65.906 


0.3 


-63.84 


0.4 


-51.4522 


-37.33 


-39.558 


0.2 


-37.774 


0.6 


-33.4955 


-26.444 


-27.996 


0.2 


-27.156 


0.8 


-24.3202 


-20.584 


-21.05 


0.1 


-21.3 



Table 8: Cs = -1.0 in SubGoals and Myopic-greedy, Ra = -1.0. 



Pu 


Average Joint Utility 


No-Comm. 


Ideal Ce = 


Comm. SubGoals - Best p 


Myopic-greedy 


0.2 


-104.925 


-62.872 


-69.286 


0.1 


-68.948 


0.4 


-51.4522 


-37.33 


-40.516 


0.1 


-40.594 


0.6 


-33.4955 


-26.444 


-28.192 


0.1 


-28.908 


0.8 


-24.3202 


-20.584 


-21.118 


0.1 


-22.166 



Table 9: Cs = —10.0 in SubGoals and Myopic-greedy, Ra = —1.0. 



5. The results arc presented for the best p, found empirically. 

6. Statistical significance has been established with t-test. 
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When Cs = — 1 (see Table 8) the utiUties attained for the Myopic- greedy approach 
when Pu < 0.8 are significantly greater than the results obtained in the heuristic case. 
When Pu = 0.8, the heuristic case was found to be better than Myopic-greedy for the best 
choice of p (Myopic-greedy obtained -21.3, and the SubGoals with p = 0.1 attained -21.05 
(variance=2.18)). The utilities attained by the Myopic-greedy agents, when Cs = —10 (see 
Table 9) and P„ in {0.2,0.4}, were not significantly different from the SubGoals case for 
the best p with significance levels 61% and 82%, respectively. However, the heuristic case 
yielded smaller costs for the other values of Pu = {0.6, 0.8}. One important point to notice 
is that these results consider the best p found for the heuristic after trying a set of discrete 
values for p (see the x-axis in Figure 5). In general trying and tuning a heuristic parameter 
can be very time consuming and the best choice may not be known ahead of time to the 
designer. On the other hand, the Myopic-greedy approach does not require any tuning of 
any parameter. In all the settings tested. Myopic-greedy always attain utilities higher than 
those attained in the SubGoals case with the worst p. 





Avc^r 


ag'(^ Communication Acts Pcirformcd 


Pa 


No-C'oniiii. 


I(k>al (\ = 


SubCioals 


AI\-opi(:-,u,T(xxly 


0.2 





31.436 


5.4 


21.096 


0.4 





18.665 


1 


11.962 


0.6 





13.426 


1 


8.323 


0.8 





10.292 


1 


4.579 



Table 10: Cs = -0.10, Ra = -1.0. 





Average Communication Acts Performed 


Pu 


No-Comm. 


Ideal Ce = 


Comm. SubGoals 


Myopic-greedy 


0.2 





31.436 


1.194 


6.717 


0.4 





18.665 


1 


3.904 


0.6 





13.426 


1 


2.036 


0.8 





10.292 





1.296 



Table 11: Cs = -1.0 in Myopic-greedy and SubGoals, Ra = -1.0. 



Pu 


Average Communication Acts Performed 


No-Comm. 


Ideal Ce = 


Comm. SubGoals 


Myopic-greedy 


0.2 





31.436 





0.416 


0.4 





18.665 





0.417 


0.6 





13.426 





0.338 


0.8 





10.292 





0.329 



Table 12: Cs = -10.0 in Myopic-greedy and SubGoals, Ra = -1.0. 



Tables 10, 11 and 12 present the average number of communication acts performed in 
each one of these cases. 
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Figure 6: The complexity of solving Dec-MDPs. 



8. Discussion 

Solving optimally decentralized control problems is known to be very hard. Figure 6 sum- 
marizes the complexity results (the rectangles stand for optimal solutions while the paral- 
lelograms stand for solutions proposed in the framework of communication-based decom- 
position mechanisms). This taxonomy helps us understand the characteristics of different 
classes of decentralized control problems and their effect on the complexity of these problems. 
The Coverage-set (Becker et al., 2004), OptlGoal and OptNGoals (Goldman &; Zilberstein, 
2004a) were the first algorithms to solve optimally some non-trivial classes of Dec-MDPs. 

This paper presents communication-based decomposition mechanisms as a way to ap- 
proximate the optimal joint solution of decentralized control problems. This approach is 
based on two key ideas: (1) separating the questions of when to communicate and what to do 
between communications, and (2) exploiting the full observability of the global states after 
each communication to generate individual behaviors that the agents will follow between 
communications. CommTinication between the decision makers serves as a synchronization 
point where local information is exchanged in order to assign an individual behavior to each 
controller. This addresses effectively applications in which constant communication is not 
desirable or not feasible. Many practical reasons could prevent agents from communication 
constantly. Communication actions may incur some costs that reflect the complexity of 
transmitting the information, the utilization of limited bandwidth that may be shared with 
other applications, or the risk of revealing information to competitive parties operating in 
the same environment. Our communication-based decomposition mechanism divides the 
global problem into individual behaviors combined with communication acts to overcome 
the lack of global information. 
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We formalized communication-based decomposition mechanisms as a decentralized semi- 
Markov process with communication (Dcc-SMDP-Com). We proved that solving opti- 
mally such problems with temporally abstracted actions is equivalent to solving optimally 
a multi-agent MDP (MMDP).We adapted the multi-step backup policy iteration algorithm 
to the decentralized case that can solve the Dec-SMDP-Com problem optimally. This algo- 
rithm produces the optimal communication-based decomposition mechanism. To provide 
a tractable algorithm, we can restrict the set of individual behaviors that are allowed. We 
proposed the LGO-MSBPI polynomial-time algorithm that computes the assignment of a 
pair of local goals and period of time k with the highest value to each possible global state. 
Adding local goals to the model seems more natural and intuitive than computing local 
behaviors based on general options. It is easier to state when a local behavior is completed 
when some local goal is reached, rather than stating sequences of local actions that even- 
tually should achieve some desired global behavior. Furthermore, an unrestricted set of 
options is larger and therefore it is computationally cheaper to compute the decomposition 
mechanism when local goals are assumed. This intuition is confirmed by our experiments. 
But the general question remain open, namely to determine when it is beneficial to compute 
local behaviors out of general options rather than assuming local goals. 

The paper also presents a simpler approximation method. It assumes that a certain 
mechanism is given, i.e., human knowledge is incorporated into the model to provide agents 
with individual policies of actions (not including communication acts). A greedy- approach 
is presented that computes the best time to communicate assuming there is only one op- 
portunity for exchanging information. The paper concludes with an empirical assessment 
of these approaches. 

In summary, this paper contributes a communication-based decomposition mechanism 
that can be applied to many of the hard decentralized control problems shown in Figure 6. 
This approach enables us to compute tractable individual behaviors for each agent together 
with the most beneficial time to communicate and change these local behaviors. The ana- 
lytical results in the paper support the validity of the approach with respect to Dec-MDPs 
with independent transitions and observations. However, it is straightforward to apply the 
approach to general Dec-POMDPs or Dec-MDPs with dependent transitions or observa- 
tions, and we believe it offers a viable approximation technique for these problems as well. 
Our approach is scalable with respect to the number of agents since all the complexity 
results presented will increase linearly as more agents are added to the system. Exchange of 
information is assumed to be via a broadcast to all the other agents. An interesting future 
extension is to study how agents can efficiently choose partners for communication to avoid 
global broadcasting. 
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